Automated AWS Resource Compliance Monitor
A scalable, serverless solution leveraging AWS services to enforce security and cost compliance across cloud resources
Project Overview
Designed and implemented a serverless compliance monitoring system that scans AWS resources, detects issues, and remediates them automatically while providing real-time reporting to stakeholders. This architecture addresses challenges in maintaining cloud security, optimizing costs, and reducing human error, all while offering scalability and automation.
Workflow
Event Triggering
EventBridge schedules compliance scans daily or dynamically based on resource changes.
Compliance Scanning
- Lambda function (compliance_scanner.py) identifies violations in S3 buckets, security groups, and EBS volumes
- Findings are stored in DynamoDB and logged in CloudWatch for metrics aggregation
- Critical findings are sent to stakeholders via SNS notifications
Violation Handling
- DynamoDB Streams trigger the Remediation Lambda function to auto-resolve common compliance issues
- Updates violation status in DynamoDB to REMEDIATED
Report Generation
Lambda function generates daily compliance summaries, uploads reports to S3, and sends report links via SNS emails
Metrics & Dashboards
CloudWatch Insights queries and dashboards visualize compliance trends, top violations, and remediation actions
Key Features
- Automated Compliance Checks: Detect violations in S3, EC2 (security groups), and EBS volumes using modular rule checks
- Real-Time Remediation: Automatically fix common issues such as enabling encryption and blocking public access
- Comprehensive Reporting: Daily compliance summaries stored in S3 and shared via email notifications
- Transparent Monitoring: CloudWatch dashboards provide insights into compliance trends and issues
- Scalable & Serverless: Uses AWS Lambda for event-driven workflows, ensuring scalability and cost-efficiency
Technical Implementation
The solution architecture implements a fully serverless approach with several key components:
1. Compliance Scanner
- Script: compliance_scanner.py
- Functionality: Scans AWS resources for compliance violations
- Logs findings in DynamoDB and CloudWatch Logs
- Alerts stakeholders of critical issues using SNS
2. Remediation
- Script: remediation.py
- Functionality: Triggered by DynamoDB Streams for real-time updates
- Auto-remediates issues such as missing S3 bucket encryption
- Updates violation status in DynamoDB to REMEDIATED
3. Reporting
- Script: reporting.py
- Functionality: Queries DynamoDB for compliance history
- Generates JSON reports stored in S3
- Sends email notifications to stakeholders with a link to the report
4. Metrics and Dashboards
- Tools: CloudWatch Insights, Custom Dashboards
- Functionality: Aggregates compliance scores, issue trends, and remediation actions into visual dashboards
Results & Impact
- Automated detection and remediation reduced manual effort by 75%
- Real-time alerts ensured swift action on critical violations
- Transparent dashboards improved visibility into compliance trends and reduced risks
- Scalable architecture accommodated growth in monitored AWS resources without additional cost
Project Gallery
Future Enhancements
- Broaden Resource Scope: Add compliance checks for IAM roles, RDS databases, and Lambda configurations
- Machine Learning Integration: Use Amazon SageMaker to predict and prevent violations based on historical trends
- Interactive Dashboards: Integrate Amazon QuickSight for advanced visualizations and analytics
- Custom Rule Library: Build an extendable library for user-defined compliance rules